SYM-AM- 15-098 


PROCEEDINGS 

OF  THE 

TWELFTH  ANNUAL  ACQyiSITION 
RESEARCH  SYMPOSIUM 


THURSDAY  SESSIONS 
VOLUME  II 


OSD’s  Obligation  &  Expenditure  Rate  Goals:  An 
Examination  of  the  Factors  Contributing  to  the 

Interference 

Robert  Tremaine,  DAU 
Donna  J.  Seligman,  DAU 

Published  April  30,  2015 


Disclaimer:  The  views  represented  in  this  report  are  those  of  the  author  and  do  not  reflect  the  official  policy 
position  of  the  Navy,  the  Department  of  Defense,  or  the  federal  government. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OE  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


Form  Approved 
0MB  No.  0704-0188 


Report  Documentation  Page 


Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  0MB  control  number. 


1.  REPORT  DATE 

30  APR  2015 

2.  REPORT  TYPE 

3.  DATES  COVERED 

00-00-2015  to  00-00-2015 

4.  TITLE  AND  SUBTITLE 

OSD’s  Obligation  &  Expenditure  Rate  Goals:  An  Examination  of  the 
Factors  Contributing  to  the  Interference 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Defense  Acquisition  University  West  Region,33000  Nixie  Way  Bldg 

50,San  Diego,CA,92147-5117 

8.  PEREORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

10.  SPONSOR/MONITOR’S  ACRONYM(S) 

11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 


12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

14.  ABSTRACT 

Managing  DoD  acquisition  programs  is  a  complicated  process.  The  turbulence  created  by  funding 
instability  makes  it  even  more  difficult.  To  help  program  offices  maintain  their  overall  funding  execution 
pace,  the  Office  of  the  Secretary  of  Defense  (OSD)  instituted  Obligation  and  Expenditure  rate  goals  over 
two  decades  ago.  Acquisition  program  managers  have  found  it  difficult  to  meet  established  Obligation  and 
Expenditure  rate  goals.  For  the  purposes  of  this  study  (sponsored  by  Nancy  Spruill,  director  of  Acquisition 
Resources  and  Analysis,  Office  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and 
Logistics)  and  based  on  Defense  Acquisition  University  and  OSD  subject  matter  expertise,  the  authors 
looked  closely  at  potential  causal  factors  that  could  be  interfering  with  the  achievement  of  these  goals.  Two 
hundred  and  twenty-nine  DoD  personnel  responded  to  a  comprehensive  survey.  The  respondents  were 
comprised  of  program  office  personnel  (program  managers  [PMs],  deputy  PMs,  budget  and  financial 
managers  [FMs],  and  contracting  officers);  program  executive  officers  (PEOs)  and  their  chief  financial 
officers;  and  a  variety  of  senior  OSD  staff  including  Headquarters  EM  senior  staff  and  Senior  Acquisition 
Executive  staff.  The  respondents  were  asked  if  they  found  metrics  helpful  in  better  meeting  OSD  goals  as 
well  as  the  use  of  any  process  improvements. 


15.  SUBJECT  TERMS 


16.  SECURITY  CLASSIEICATION  OE: 

17.  LIMITATION  OE 
ABSTRACT 

18.  NUMBER 
OE  PAGES 

19a.  NAME  OE 
RESPONSIBLE  PERSON 

a.  REPORT 

unclassified 

b.  ABSTRACT 

unclassified 

c.  THIS  PAGE 

unclassified 

Same  as 
Report  (SAR) 

19 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


The  research  presented  in  this  report  was  supported  by  the  Acquisition  Research 
Program  of  the  Graduate  School  of  Business  &  Public  Policy  at  the  Naval 
Postgraduate  School. 

To  request  defense  acquisition  research,  to  become  a  research  sponsor,  or  to  print 
additional  copies  of  reports,  please  contact  any  of  the  staff  listed  on  the  Acquisition 
Research  Program  website  (www.acquisitionresearch.net). 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


OSD’s  Obligation  &  Expenditure  Rate  Goals:  An 
Examination  of  the  Factors  Contributing  to  the 

Interfe  rence 


Robert  Tremaine — is  associate  dean,  Outreach  and  Mission  Assistance,  Defense  Acquisition 
University  West  Region.  Coi.  Tremaine  is  a  retired  Air  Force  coionel  and  has  over  26  years  of 
experience  in  air,  missile,  and  space  weapons  systems  acquisitions.  He  hoids  a  BS  from  the  U.S.  Air 
Force  Academy  and  an  MS  from  the  Air  Force  Institute  of  Technology.  Col  Tremaine  is  Level  III 
certified  in  both  program  management  and  systems  planning,  research,  development,  and 
engineering,  [robert.tremaine@dau.mil] 

Donna  Sellgman — is  a  management  information  systems  manager  at  the  Defense  Acquisition 
University  West  Region.  She  has  extensive  experience  with  developing  complex  business  knowledge 
applications,  performing  system  analyses,  and  conducting  research.  She  has  also  supported  legacy 
code  and  reengineering  efforts  of  other  major  business  system  applications.  Seligman  holds  a  BS  in 
Information  Decision  Systems  from  San  Diego  State  University,  [donna.seligman@dau.mil] 

Abstract 

Managing  DoD  acquisition  programs  is  a  complicated  process.  The  turbulence  created  by 
funding  instability  makes  it  even  more  difficult.  To  help  program  offices  maintain  their  overall 
funding  execution  pace,  the  Office  of  the  Secretary  of  Defense  (OSD)  instituted  Obligation 
and  Expenditure  rate  goals  over  two  decades  ago.  Acquisition  program  managers  have  found 
it  difficult  to  meet  established  Obligation  and  Expenditure  rate  goals.  For  the  purposes  of  this 
study  (sponsored  by  Nancy  Spruill,  director  of  Acquisition  Resources  and  Analysis,  Office  of 
the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics)  and  based  on 
Defense  Acquisition  University  and  OSD  subject  matter  expertise,  the  authors  looked  closely 
at  potential  causal  factors  that  could  be  interfering  with  the  achievement  of  these  goals.  Two 
hundred  and  twenty-nine  DoD  personnel  responded  to  a  comprehensive  survey.  The 
respondents  were  comprised  of  program  office  personnel  (program  managers  [PMs],  deputy 
PMs,  budget  and  financial  managers  [FMs],  and  contracting  officers):  program  executive 
officers  (PEOs)  and  their  chief  financial  officers;  and  a  variety  of  senior  OSD  staff  including 
Headquarters  FM  senior  staff  and  Senior  Acquisition  Executive  staff.  The  respondents  were 
asked  if  they  found  metrics  helpful  in  better  meeting  OSD  goals  as  well  as  the  use  of  any 
process  improvements. 

Introduction 

In  the  months  preceding  this  research  effort,  Nancy  Spruill,  director,  Acquisition 
Resources  and  Analysis,  Office  of  the  Under  Secretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics  (OUSD  [AT&L])  solicited  support  from  the  Defense  Acquisition 
University  (DAU)  to  help  uncover  the  causal  factors  that  could  be  interfering  with  the 
attainment  of  the  Office  of  the  Secretary  of  Defense’s  (OSD’s)  Obligation  and  Expenditure 
rate  goals.  To  learn  more  about  the  intervening  obstacles,  DAU,  along  with  assistance  from 
the  OSD,  developed  a  comprehensive  survey  that  queried  experienced  and  high-level 
Department  of  Defense  (DoD)  personnel  Involved  in  a  weapon  program’s  decision  chain. 
What  we  learned  from  the  subsequent  analysis  confirmed  several  previous  suspicions.  The 
data  also  indicated  the  prevalence  of  more  underlining  perception  variances  among  many  of 
the  factors  that  could  be  undermining  program  execution  itself. 
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Research  Methodology 

Two  hundred  and  twenty-nine  DoD  personnel  responded  to  this  survey.  The 
respondents  were  comprised  of  program  office  personnel  (program  managers,  deputy 
program  managers,  budget  and  financial  managers,  and  contracting  officers),  program 
executive  officers  and  their  chief  financial  officers  (CFOs),  and  a  variety  of  senior  staff  at  the 
OSD  including  Headquarter  Financial  Management  (FM)  senior  staff  and  Senior  Acquisition 
Executive  (SAE)  staff  (Table  1).  Because  several  functional  areas  saw  lower  response  rates, 
a  more  detailed  analysis  of  the  causal  factors  was  restricted  to  an  aggregate  sample  size 
given  the  confidence  levels  required  to  draw  any  inferences  or  conclusions. 


Table  1.  Respondent  Demographics 


SURVEY  RESPONDENT  DETAILS 

ACAT  LEVELS 

Respondent 

Groups 

TOTALS 

Respondent 

Distribution 

1 

II 

111 

Program 

Olncc^ 

P€0* 

Senlof 

Staffs 

Responses 

Queried 

Response 

fUte 

Total 

91 

2a 

23 

142 

63 

24 

229 

698 

33% 

'  IriCfu^  From  Jll  CompOfi«fiti  and  several  OoO 

^  Proflfam  Deputy  Program  Managers,  Budget  and  F^nanctal  Managers  (BFM),  Deputy  Bf Ms, 

an oContracting  Officers 

*  PfOf  rarn  Evecutivr  OfffcefSfPEOsK  Deputy  PEOs  and  their  Chief  Financial  Officers 

*  Meadpuarter  Firsirvcial  Managers  arvd  Senior  Acquisition  EwecuUve  Staff 


Respondents  ranked  the  impact  of  64  factors  under  nine  categories  (Figure  1).  The 
researchers  then  assessed  the  rankings  using  a  top  box  (TB)  three  methodology  (i.e.,  the 
percentage  of  5,  6,  and  7  responses  on  a  Likert-like  scale  from  1-7).  Since  the  frequency  of 
occurrence  for  some  factors  could  also  be  contributing  to  the  interference,  the  researchers 
included  an  additional  selection  (e.g.,  daily,  weekly,  monthly,  etc.)  to  isolate  any  potential 
ignition  areas. 
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Figure  1.  Factor  Categories 


Discussion 
Factor  Distribution 

Figure  2  shows  the  distribution  of  all  64  factors  assessed.  Three  factors  reported  an 
impact  rating  of  two  standard  deviations  above  the  mean  (denoted  by  +2a);  six  factors 
reported  an  impact  rating  of  one  standard  deviation  above  the  mean  (denoted  by  +lcr);  and 
22  factors  rose  above  an  average  impact  rating  (denoted  by  x).  The  remaining  33  factors  fell 
below  the  aggregate  x. 

Nineteen  of  the  22  factors  measured  for  frequency  of  occurrence  resulted  in  an 
impact  rating  above  39%.  Sometimes,  just  one  occurrence  appeared  to  have  a  significant 
impact. 


-  155- 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  EORINEORMED  CHANGE 


25 


20 


t 

o 

t5  15 

'S 

1 10 

3 


Impact  Factor  Distribution 


\ 


♦  lCI%-249t  ♦  259t-38%  ♦  39%-52«  ♦  53%-66K  ♦  67%-69% 

-2<j  -lo  X  4 1<T  +2a 

_ imtxet  ftatinet _ 


Figure  2.  Factor  Ranking  Distribution 
The  Causal  Factors  Rank  Ordered 

Table  2  lists  the  relative  ranking  of  all  64  factors  in  the  context  of  top  box  descending 
order  to  provide  a  comprehensive  view  of  all  factors,  although  the  remaining  discussion  in 
this  paper  addresses  the  factors  above  x.  Unrealistic  Spend  Plans  (F^°),  also  one  of  the 
factors  assessed,  is  generally  valued  as  a  written  forecast  of  a  program’s  funding  needs  and 
establishes  Obligation  and  Expenditure  projections.  However,  spend  plans  are  subjected  to 
so  many  real  world  eventualities  that  updating  them  becomes  problematic  in  sustaining  its 
forecasting  value. 
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Table  2.  Impact  Factor  Ratings  in  Aggregate  Descending  Order 


Factors  Rated  by  Adverse  Impact 

TB 

X 

a 

f  1  or  tM  aiJhorty  due  to  CoftnuvH)  Ausnorty 

ft9S 

629 

241 

f7  Cortraci  negoftatom  dNays 

87% 

606 

269 

F3  Coiaraci  award  dHa>s 

67% 

600 

266 

F4  d  CcnractnQ  CtlKtn 

64S 

4  79 

266 

F5  CongmsonH  mar% 

61% 

487 

286 

F5  C«r«ractor  pfopoaal  prep  dMtfrft 

00% 

4  87 

269 

F  7  OSO  Creeled  RMO  adlfusPnerc 

68% 

460 

263 

F8RfPprepdt«ays 

67% 

463 

246 

66% 

liH 

261 

F 10  Urveakabc/oyedy  opimstic  tperv)  pyns 

62% 

430 

244 

F 1 1  d^anpea  n  trtrr  requaements 

61% 

4  16 

2  43 

Fl?CKanpesto  program  ac<Maca)n  ^aiegy 

61% 

441 

262 

F 13  Chariges  f)  other  suke^otder  reguaemerts 

60% 

432 

234 

F 14  Preparaig  DAE  level  review  ara)  decisvn 

60% 

4  16 

218 

F 1 S  Lack  of  decision  audionty  at  expected  leveH 

60% 

422 

262 

'  FlOImplemcniation  oInewOSO.'Stmcapoacy 

49% 

420 

2S9 

F 1 7  Cornporiera  daected  POM  adiMflr^ 

49% 

426 

2  61 

F 18  Awacaig  reprogrammng  acbon 

49% 

423 

2  44 

F19Changesai  userpnortes 

47% 

400 

238 

F20  ReaAsbc  spend  pMm  tui  nsfcs  mMenAMed 

46% 

400 

2  21 

F71  Program  diiays  from  adcMiooil  deveiopmert  testaig  or  other  prcregiaste  events 

44% 

409 

236 

F22  DCAA  admaetfrafrve  acbom 

44% 

392 

261 

F23  unplanned  Congreswial  adds  to  P8  request 

43^ 

390 

2  41 

F24  Use  of  la^dcMiMd  contract  action  dHays 

42% 

3  73 

266 

F2S  Expendture  corttngtnl  on  harctware  ddMery 

41% 

392 

241 

F3ft  lots  of  taxing  ftou^  reprogranaiag  adontoti^  prvxfy  frr^awTvts  loPEOpordblD 

41% 

389 

246 

f27  Lack  of  Experience  ievets  n  key  acqivsaionfraicfronil  areas 

40% 

390 

230 

F28  AwaCaig  DAE  level  rewcw  and  deepen 

40% 

360 

2  42 

F29  Shortage  Of  Co«  Efimattrt 

40% 

367 

2  37 

F30  Shortage  of  Busaasstxianct  pmomi 

38% 

3  73 

230 

F31  fVoarammatic  c  orfkcts  berween  ocM  andewme  ccycracior 

i9S 

212 

F32  Preparaig  SAE  CAE  level  renew  and  decavon 

38%  ; 

3  74 

202 

F33  Delays  aicortraciorpaymert  due  lo  late  awoces 

37% 

367 

236 

F34  Unohkgiled  prva  year  liavkig  not  adegiiMaCy  factored 

38%  ' 

3  67 

223 

F  36  Cornponeni  CempiroAar  VViM^ 

>6% 

358 

234 

F36  OCMA  admavstrative  acfrorts 

36% 

3  42 

236 

F37  RedaecOon  of  confiracior  cdoris 

36% 

347 

223 

F38  OSO  Comptroiw  Withhold 

34% 

343 

2  37 

F  39  Shortage  of  Tec  rvEng^Test  persormH 

34% 

351 

217 

F40  Shortage  of  Audiors 

33% 

317 

2  43 

F41  Slower  bum  rate  than  expected  due  to  urdavorable  Sf*l 

33% 

326 

214 

F42  Awaftng  SAE/CAE  level  review  and  dectMn 

32% 

333 

230 

F43  SAE/CAE/Componert  drected  reprogrammaig 

32% 

327 

230 

'  F44RecfSsa)n 

32% 

316 

246 

F  46  Changes  ffi  systems  spec  s 

31% 

330 

203 

F46  Teniae  of  PM  and  others  ai  key  posaons 

31% 

311 

218 

F47  Hofrkng  awardVKentrve  fees  at  commtmenllor  Mwae  obigabon 

29% 

323 

236 

F48  fenadenuate  tranng 

29% 

329 

213 

F49  Shortage  of  Managers 

28% 

310 

217 

F50  Insiifrcerth  planned  OCOKavSng 

27% 

307 

2  27 

F61  Sheatage  of  sun 

26% 

299 

212 

FS2  Coidractor  rework 

26% 

300 

214 

F63DiS«fredparmrm  fra  schedtAngegraq  tees  troyesscavriqts^pqlgwwxebMBdpaTiert^ 

26% 

308 

220 

^  Effect  of  contract  (ype  on  ou^  rates 

24% 

2  17 

F66MateneirSystefTisCorT¥TtendConyfrolteWlgrhoid 

2/1 

217 

F66Awacaig  PEO  level  review  anddecison 

24% 

280 

201 

FS7  Termaidbon  luibdty 

27% 

2  72 

217 

f  56  InsiibctefC  workplace  looiyapps 

22% 

282 

2  01 

F69  PEO  drected  programmng 

21% 

283  ' 

2  10 

F60  Slower  bum  rale  ffian  expected  due  lo  fevorabie 

21% 

2  77 

196 

F€i  PEOWittaidd 

20% 

238 

199 

F62  F^epwig  PEO  tewf  review  and  dectepn 

20% 

266  ' 

163 

Fd3  Production  me  issues 

19% 

282 

208 

Fd4Labor  dsputes 

10% 

188 

164 

+  67% 


+  ^  53% 


ji:^35% 


~\oz2S% 
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Factors  and  Respondent  Groups 

Figure  3  accounts  for  the  31  factors  above  the  mean  and  by  respondent  group  seen 
in  Table  1 .  The  31  factors  were  the  only  ones  further  evaluated  in  this  study  unless  a  factor 
shifted  above  x  after  any  further  delineation  (e.g.,  ACAT  levels,  military  components, 
position,  etc.).  Unexpectedly,  the  individual  factors  showed  widespread  perception 
disparities  among  the  respondent  groups  for  the  factors  that  fell  below  +2a.  After  analyzing 
the  specific  individual  factors  among  all  the  respondent  groups,  seven  of  the  31  factors  had 
an  unusually  large  a.  As  a  result  of  these  conspicuous  gaps,  we  turned  to  the  qualitative 
data.  We  also  watched  for  any  strong  correlations  (e.g.,  correlation  coefficients  (r)  >  0.7)  to 
better  understand  the  reasons  for  the  differences,  as  well  as  the  influence  of  any  intervening 
and/or  moderating  factor  couplings.  The  remaining  discussion  addresses  the  31  impact 
factors  in  descending  order  from  highest  to  lowest. 


ism 


Figure  3.  Impact  Factor  Ratings  Above  x  in  Aggregate  Descending  Order  With 

Respondent  Group  Low  and  High  Ratings 

The  Factors  That  Ranked  Above  +  2a 

In  Figure  3,  late  release  of  full  obligation/budget  authority  due  to  Continuing 
Resolution  Authority  (F^),  Contract  Negotiations  Delays  (F^),  and  Contract  Award  Delays 
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(F^)  all  rose  above  la,  where  67%  or  more  of  the  respondents  claimed  they  had  the  highest 
adverse  impact  of  all  factors  measured.  The  occurrence  of  CRA  had  the  most  significant 
negative  impact  to  Obligation  and  Expenditure  rates.  It  also  had  one  of  the  smallest 
variances  {a)  among  the  respondent  groups.  Even  with  the  expectation  that  CRA  might 
prevail  and  the  subsequent  planning  that  followed  for  such  a  likely  event,  many  PMs  pointed 
to  an  overly  conservative  and  slow  internal  vetting  process  posture  that  created  additional 
obstacles  in  meeting  OSD  goals.  Several  PMs  recommended  the  use  of  some  sort  of  “CRA 
variable”  to  temporarily  offset  the  consequences  of  CRA  If  the  required  funds  were  not 
released  as  originally  projected.  Next  in  rank  order  were  contract  negotiations  and  contract 
award  delays.  The  respondents  emphasized  that  the  DoD  could  fix  the  problem  more  readily 
since  unlike  CRA,  these  factors  were  under  internal  control.  When  asked  what  could  be 
done  to  reduce  the  adverse  effects  of  all  three  factors,  the  respondents  recommended  the 
“Inclusion  of  more  risk  mitigation  into  contract  award  planning,  more  realistic  timelines,  more 
realistic  plans,  greater  funding  stability,  reduction  in  bureaucratic  obstacles,  more 
synchronized  internal  processes,  and  better  aligned  accounting  systems.” 

The  Factors  That  Ranked  Above  +la 

This  next  line  of  demarcation  (Figure  3,  factors  F'^-F®)  included  many  contracting- 
related  factors  (i.e..  Shortage  of  Contracting  Officers  [F^],  Contractor  Proposal  Prep  Delays 
[F®],  RFP  Prep  Delays  [F®],  and  Source  Selection  [F®]).  Nearly  all  the  factors  showed  the 
emergence  of  a  more  alarming  a  between  the  individual  respondent  groups — as  high  as 
18%  in  one  case  (i.e..  Proposal  Prep  Delays  [F®]).  For  this  particular  factor,  PCOs  reported 
the  highest  impact  while  PMs  ranked  it  as  the  lowest.  Senior  staff  cited  that  Shortage  of 
Contracting  Officers  (F^)  created  the  highest  impact,  while  PCOs  reported  it  had  the  lowest 
impact.  With  a  7%  a,  it  was  the  lowest  among  all  six  factors  in  this  grouping. 

Given  that  six  of  the  top  nine  factors  were  contract-specific  factors  that  ranked  above 
+1a  (see  Figure  3),  it  came  as  little  surprise  to  see  so  many  reinforcing  comments  surface: 

•  “Lack  of  experienced  and  qualified  contract  specialists” 

•  “Alarmingly  low  personnel  qualified  ...  many  unsure/lack  guidance  and 
experience” 

•  Significantly  stressed  with  overtime  to  complete  all  contracting  actions  prior  to 
close  of  fiscal  year” 

•  “Inadequate  training  . . .  inordinate  number  of  interns  with  very  low  experience 
in  all  career  fields” 

•  “Lack  of  sufficient  legal  personnel  trained  in  acquisition” 

•  “Loss  in  brain  trust  and  skill  to  develop  complete,  clear  SOWs  using  proactive 
contract  language” 

•  “SOW  writing  and  the  teaching  of  SOW-writing  classes  is  greatly  left  to 
contractors  or  support  contractors  resulting  in  unclear  language” 

The  highest  frequency  of  occurrence  was  also  associated  with  contracting-related 
factors  (Figure  3).  By  far,  the  aggregate  respondents  rated  Shortage  of  Contracting  Officers 
(F^)  as  the  single  highest  factor  among  all  22  factors  measured  for  frequency.  Because  the 
contracting  activity  timeline  generally  has  lengthy  durations,  any  disruption  appears  to  have 
an  unmistakable  impact  on  contract  award.  Shortage  of  Contracting  Officers  (F'^)  was  seen 
as  having  the  most  significant  impact.  Several  respondents  said  that  “multiple  contracting 
actions  were  having  compounding  consequences.” 
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The  two  remaining  factors  above  +1a,  Congressional  Marks  (F^)  and  OSD  Directed 
RMD  Adjustment  (F^),  had  very  low  frequency  of  occurrences  but  still  reported  a  very  high 
impact,  similar  to  CRA.  When  combining  these  with  F^,  all  three  appear  to  be  a  strong 
antecedent  force  (or  moderating  factor)  to  the  already  time-consuming  chain  of  contracting 
actions. 


The  Factors  That  Ranked  Above  x 

This  final  grouping  (Figure  3,  factors  F''°-F^'')  accounted  for  the  remaining  22  impact 
factors.  Perception  polarities  persisted  especially  between  two  respondent  groups — senior 
staff  outside  the  program  office  and  PMs  inside  program  offices.  For  PMs  in  every  case 
except  one  (i.e.,  Component  Directed  POM  Adjustment  [F"'^]),  the  impact  factors  ranked  well 
below  X.  In  sharp  contrast,  senior  staff  in  every  case  except  one  (i.e..  Component  Directed 
POM  Adjustment  [F^^])  stated  the  majority  of  top  31  factors  had  the  largest  impact  or  close 
to  it  among  all  respondent  groups. 

Even  though  the  remaining  impact  factors  above  x  are  still  significant,  the 
researchers  shifted  the  focus  to  the  presence  of  any  strong  correlations  since  factor 
couplings  could  be  having  a  moderating  effect  and  require  a  closer  look. 

The  Factors  That  Correlate 

Table  3  summarizes  the  strongest  and  weakest  factor  correlations  for  all 
respondents  queried.  Several  strong  correlations  surfaced  for  factors  above  x.  User 
Requirements  (F'’"')  and  User  Priorities  (F''®)  were  very  strongly  correlated.  In  three  specific 
instances,  two  factors  above  x  were  very  strongly  correlated  with  three  factors  that  fell 
below  x:  Key  Acquisition  Experience  (F^^)  and  Inadequate  Training  (F^^);  Key  Acquisition 
Experience  (F^^)  and  Tenure  of  PM  and  other  Key  Positions  (F'*®);  and  DCMA  Administration 
Actions  (F®®)  and  DCAA  Administration  Actions  (F^^).  Three  contract-related  factors  (F"^,  F®, 
and  F®)  showed  weaker  correlations  than  unexpected.  Because  a  factor  had  a  weak 
correlation  does  not  mean  it  had  any  less  importance,  but  any  course  of  action  intended  to 
mitigate  the  presence  of  any  impact  factor  strongly  correlated  with  another  should  be 
weighed  more  heavily  in  any  recommended  action.  For  example,  the  turnover  of  PMs  could 
be  part  of  the  experience  quotient. 


Table  3.  Factor  Correlation  Couplings 
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Factor  Plotting 

The  researchers  generated  a  scatter  plot  diagram  (Figure  4)  that  punctuated  how  the 
31  factors  fluctuated  between  impact  and  frequency  of  occurrence.  In  some  cases,  the 
impact  of  certain  factors  occurred  with  low  frequencies  of  occurrence.  In  other  cases,  the 
frequency  of  occurrence  compounded  the  impacts. 

The  research  data  was  rebased  to  a  Likert-like  scale  for  plotting  the  frequency  and 
adverse  impact  response  averages.  The  researchers  included  Factors  F^^-F^^  in  Figure  4 
because  they  only  fall  slightly  below  x. 
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Figure  4.  Scatter  Plot  of  Impact  Factors  With  Frequency 

For  the  relationships  that  were  co-linear  (e.g.,  the  most  strongly  correlated  depicted 
in  Table  3),  the  researchers  explored  whether  they  also  behaved  as  strong  predictors  across 
the  sample  population.  After  investigating  t-ratios  (used  with  ACAT  level  factors)  and  beta- 
weights  (used  for  the  sample  population),  we  determined  that  the  relationships  were  not 
significantly  co-linear  enough  to  substantiate  causation.  Consequently,  there  was  no  merit  in 
running  any  further  regression  that  analyzed  the  factors  as  predictors.  However,  the 
researchers  conducted  another  set  of  tests  by  modulating  certain  respondent  demographics 
and  holding  x  constant. 

Factor  Plotting — Modulating  ACAT  Leveis 

Figure  5  shows  how  the  factor  rankings  changed  after  isolating  ACAT  levels. 

ACAT  I 

Funding  and  requirements  factors  (F^®,  F^®,  F^®,  and  F®®)  previously  ranked  above  x 
dropped  below  x,  while  Contractor  Proposal  Delays  (F®)  rose  markedly  to  become  the 
highest  impact  factor.  Component  Directed  POM  Adjustment  (F^^)  made  a  noticeable  shift  to 
the  top  nine  factors  (or  one  standard  deviation  above  the  mean). 
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ACATII 

Fifteen  of  the  factors  previously  ranked  above  x  dropped  below  x  (leaving  only  F\  F^, 
F^,  and  F^^).  Four  of  the  factors  that  fell  below  x  included  contracting-related  factors  (F^,  F®, 
F®,  and  F®). 

ACAT  III 

Six  of  the  factors  (F^®,  F^®,  F^®,  F®\  F®®,  and  F®^)  previously  ranked  above  x  dropped 
below  X.  Shortages  of  Personnel  (F®®,  F®®,  F®®,  and  F®^)  and  Redirection  of  Contractor  Efforts 
(F®^)  became  more  dominating  issues  for  the  respondents.  Changes  in  User  Priorities  (F^®), 
Changes  in  Stakeholder  Requirements  (F^®),  and  Funding  Loss  from  Reprograming  Actions 
due  to  Fligher  Priority  Requirements  (F®®)  all  moved  significantly  above  x. 

This  more  detailed  differentiation,  as  found  in  the  scatter  plots,  gives  additional 
insight  into  ACAT -specific  areas  through  a  more  granular  view  of  the  factors  that  would 
benefit  from  a  more  focused  investigation.  In  some  cases,  reducing  frequency  of  occurrence 
or  perhaps  instituting  more  early  warning  metrics  could  have  a  marked  effect  in  reducing 
any  adverse  impacts. 
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Figure  5.  Factor  Ratings  >3c  by  ACAT  Level 

Factor  Plotting — Modulating  Respondent  Groups 

Figure  6  shows  how  the  factor  rankings  changed  after  isolating  the  Respondent 
Groups. 

Program  Office 

Six  factors  dropped  below  x:  Awaiting  Reprogramming  Action  (F^®),  Changes  in  User 
Priorities  (F^®),  Program  Delays  from  Prerequisite  Events  (F®^),  Unplanned  Congressional 
adds  to  PB  Request  (F®®),  Use  of  Undefinitized  Contract  Action  Delays  (F®"^),  and  Loss  of 
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Funding  through  Reprogramming  Action  to  higher  priority  Requirements  to  PEO  Portfolio 
(F^®).  No  factors  fell  below  x. 

PEO 

Use  of  Undefinitized  Contract  Action  Delays  (F^^)  fell  below  x  while  four  factors  rose 
above  x:  Shortage  of  Cost  Estimators  (F^^),  Shortage  of  Business  and  Finance  Management 
Personnel  (F^®),  Component  Comptroller  withhold  (F^®),  and  Insufficiently  Planned  OCO 
Funding  (F^°). 

Senior  OSD  Staff 

Awaiting  Reprogramming  Action  (F"'®)  fell  below  x  while  13  factors  rose  above  x. 

For  PEO  and  senior  OSD  staff,  personnel  shortages  (F^^,  F3°,  F^°)  became  more 

dominant,  while  awaiting  reprogramming  action  (F''^)  became  less  dominant  for  program 
office  and  senior  OSD  staff  personnel.  Of  the  three  grouping  in  this  particular  case,  nowhere 
were  there  more  factor  increases  than  for  senior  OSD  staff  personnel.  The  rise  in  factors 
F^"^,  F'^^,  and  F®®  seemed  intuitive  since  senior  staff  may  see  first-hand  the  longer  time  it 
takes  for  programs  to  react  to  changes  in  their  plans.  However,  it  was  very  interesting  to  see 
what  senior  OSD  staff  personnel  felt  represented  the  major  impediments  to  meeting  the 
OSD’s  Obligation  and  Expenditures  rate  goals  that  program  office  personnel  did  not, 
especially  shortage  of  personnel  and  contract-specific  factors  like  F^®  and  F^^.  This  wide 
perception  disparity  deserves  a  more  intensive  understanding  since  it  could  be  creating 
false  perceptions  that  could  lead  to  misrepresented  positions  and  even  unsubstantiated 
decisions. 
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Figure  6.  Factor  Ratings  >  x  by  Grouped  Respondent  Position 
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Respondent  Comments  Regarding  the  Factors 

The  respondents  were  also  asked  several  open-ended  questions  about  the  use  of 
metrics  they  found  that  helped  them  better  meet  OSD  goals,  as  well  as  any  process 
improvements  they  would  recommend.  They  stated  that  the  metrics  that  made  a  difference 
for  them  included  “real-time  monitoring,  frequent  reviews,  tight  coupling  to  contractor  actions 
and  milestones,  and  realistic  spend  plans  with  inch  stones.”  As  for  necessary  improvements 
to  current  processes,  the  respondents  recommended  the  inclusion  of  a  CRA  duration 
variable  that  readjusted  expectations,  establish  more  realistic  program  goals,  ensure  more 
funding  stability,  reduce  bureaucratic  obstacles  and  streamline  more  outdated  processes, 
forge  greater  cooperation  between  government  and  industry,  and  synchronize  disparate 
accounting  systems  used  in  obligation/expenditure  reporting. 

The  respondents  provided  a  number  of  qualitative  comments  that  reinforced  the 
quantitative  data,  especially  for  the  factors  above  >  x  that  were  causing  obligation  rate 
interference: 


Xi 


Buiin»s  Ops 


t]iK4ftion 


C<Hvtr*ctini 


“Takes  too  long  to  get  Acquisition  Strategies  and  Acquisition 
Plans  written  and  approved  " 

"Personnel  do  not  have  experience  with  the  subject  matter" 


“Inadequate proposals,  protracted  negotiations,  lengthy  audits, 
and  lengthy  pre-awani  processes  " 


‘Had  to  deferAe-prioritize  requirements  execution  and  cany 
forward  funding  to  cover  cutbacks  shortfall. " 

"Changes  in  requirements  precipitated  by  other  stakeholders' 
actions  and  ill-defined  requirements  ’’ 

"User  leadership  routinely  changes  requirement  d-  priorities" 

"MIPR  billing  process  can  delay  e.xpenditures  from  90  to  120 
days.  " 

"Delays  in  negotiating  best  deal  for  gov't  and  sometimes  delays  in 
getting  acceptable  proposals  " 

"Extensive  reviews,  too  long  to  get  decision  briefs  through 
oversight  layers — not  always  value  added" 

"Multiple  instances  where  milestone  documentation  took  upwards 
of  9  months  to  a  year  to  get  approved" 

“...the  problem  isn't  unrealistic  or  ovcriv  optimistic  spend  plans  as 
much  as  it’s  not  knowing  when  funds  will  he  appropriated  and  how 
much  w  ill  be  apportioned  by  the  e.vecufing  organization.  " 


Figure  7.  Sampling  of  Respondent  Comments 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  EORINEORMED  CHANGE 


-  164- 


Comparison  With  a  Simiiar  Study 

After  this  study  was  completed,  the  OUSD(AT&L),  ODCAPE,  and  OSD(C)  sponsored 
a  related  effort  with  the  Institute  for  Defense  Analyses  (IDA)  entitled  Implications  of  DoD 
Funds  Execution  Policy  for  Acquisition  Program  Management  (Conley  et  al.,  2014).  IDA  was 
asked  to  increase  the  current  understanding  of  the  extent  and  causes  of  under-execution 
and  suggest  changes  to  improve  outcomes.  They  took  a  two-fold  approach:  (1)  Examine 
trends  in  the  ability  of  the  DoD  to  execute  appropriated  funds,  and  (2)  conduct  an  in-depth 
investigation  into  the  causes  of  funds  under-execution  for  selected  programs  and  the  effects 
on  those  programs  of  associated  financial  management  practices.  After  drawing  insights 
from  an  in-depth  investigation  of  25  individual  programs  during  face-to-face  interviews,  they 
categorized  their  causal  factors  along  five  areas: 

•  Contracting  issues  (i.e.,  personnel  shortages  and  inexperience,  award 
protests,  peer  reviews  of  contracting  process  documentation,  and  negotiation 
delays) 

•  Congressional  actions  (i.e.,  additions  and  reductions  to  requested  funding, 
continuing  Resolutions  [CRs],  and  sequestration) 

•  Management  actions  and  program  events  (i.e.,  changes  to  requirements, 
contract  type,  schedule,  responses  to  operational  needs,  technical  and 
testing  problems,  and  slow  contractor  billing) 

•  Policy  choices  (i.e.,  use  of  execution  benchmarks  and  withholding  funding  by 
services  under  CRs) 

•  Program  office  personnel  shortages  and  experience  levels 

Figure  8  shows  how  the  study  results  are  very  similar,  although  IDA  did  not  measure 
“frequency.” 
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Contributing  Factors 
(Frequency  Vs  Impact) 
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Figure  8.  Comparison  of  DAU  and  IDA  Studies 
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Recommendations 

What  next?  Based  on  the  research  findings  of  this  study,  there  are  a  number  of 
impact  factors  above  ^that  if  sufficientiy  addressed  could  heip  iower  the  barriers  to  the 
attainment  of  the  OSD’s  Obligation  and  Expenditure  rate  goals.  Hence,  we  offer  the 
following  recommendations: 

•  Institute  an  Obiigation  and  Expenditure  baseline  adjustment  for  programs 
affected  by  any  funding  delay  or  limitation  (especiaiiy  CRA),  then  measure  a 
program’s  progress  to  that  revised  adjustment. 

•  More  thoroughiy  review  the  entire  contracting  action  vaiue  chain.  Look  closely 
at  efficiency  opportunities  aiong  the  review  and  decision  cycle  continuum, 
especiaiiy  from  the  time  an  RFP  is  developed  to  the  time  a  contract  is  iet.  Set 
reasonabie  time  thresholds  with  triggers  that  afford  more  proactive  measures 
by  PMs  and  confirm  productivity. 

•  Estabiish  a  recurring  communication  forum  among  key  stakeholders, 
especially  PMs  and  the  OSD,  to  dialogue  more  frequently  and  eliminate 
perception  gaps  that  could  be  creating  counterproductive  actions  and 
misconceptions. 

•  Track  requirement  changes  throughout  a  program’s  life  and  look  more 
strategically  at  the  effects  on  program  execution  and  accompanying 
Acquisition  Program  Baselines  (APBs).  Despite  ACAT  levels,  there  is  an 
obvious  ripple  effect  that  is  associated  with  any  substantive  change  in 
program  content  across  a  program’s  life  that  should  be  codified  more 
comprehensively.  However,  there  are  also  issues  associated  with  different 
ACAT  levels  which  must  be  noted. 

•  Review  the  program  review  cycle  and  streamline  wherever  possible.  Checks 
and  balances  within  the  DoD’s  acquisition  community  are  a  vital  constituent 
component  of  program  execution,  but  every  review  should  have  a  distinctive 
purpose,  exit  criteria,  and  associated  suspense  date  that  is  just  as  material 
and  credible. 

•  Build  and  maintain  realistic  spend  plans,  measure  against  them,  account  for 
contingencies,  and  make  adjustments  with  required  frequency  due  to  real 
world  realities.  Collaborate  with  senior  leadership  early  enough  about 
required  adjustments  to  avoid  more  draconian  measures  later. 

•  Validate  the  key  personnel  shortage  areas  and  recognize  the  time  it  takes  to 
rebuild  those  experience  levels. 

•  Nurture  experience  in  key  functional  areas  with  strong  catalysts  such  as 
disciplined  on-the-job  training  (OJT)  programs,  mentoring,  and  guidance. 

With  the  recent  surge  of  contracting  specialist  interns,  their  progress  as  a 
group  should  be  measured  more  carefully. 

•  Evaluate  the  real  effects  of  reprogramming  action  or  realignment  of  future 
budget  decisions  before  any  corrective  action  is  taken. 

•  Conduct  a  wholesale  review  of  the  program  execution  metrics  currently  in 
place  and  determine  their  usefulness  and  effectiveness.  What  are  they 
actually  measuring?  Consolidate  whenever  practical  and  eliminate  the  ones 
that  have  outlived  their  usefulness. 
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•  Encourage  innovation  and  avoid  the  “bookkeeping  process,”  as  RAND 
Corporation  (2009)  found  in  a  recent  study  could  be  limiting  improvements 
championed  by  PMs. 

Summary 

This  research  exposed  a  number  of  challenges  that  could  easily  be  mitigated  by 
more  frequent  communication  and  especially  a  better  appreciation  of  stakeholder 
management.  There  are  so  many  stakeholders  involved  in  the  acquisition  process.  No 
stakeholder  should  be  dismissed  without  a  more  intensive  assessment  of  their  (potential) 
contribution.  Sometimes,  either  their  voice  is  not  heard  or  their  concern  not  appropriately 
considered.  Next,  having  a  program  management  strategy  that  can  help  leaders  react  to 
funding  reductions  is  also  critical.  A  wide  variety  of  financial  tools  exist  that  track  and  predict 
funds  execution,  but  Spend  Plans  that  serve  as  the  common  device  to  convey  program 
execution  have  to  be  current  and  agile  enough  to  demonstrate  reality  and  common  sense 
for  whatever  curve  balls  come  their  way.  Finally,  as  baby  boomers  start  to  retire  at  a  more 
aggressive  rate,  experience  will  matter  even  more.  An  OJT  program  that  nurtures 
experience  and  leadership  development  as  well  as  demands  critical  thinking  is  just  what  the 
acquisition  community  should  expect. 
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